Method and Apparatus to Infer Object and Agent Properties, Activity Capacities, Behaviors, and Intents from Contact and Pressure Images

ABSTRACT

An apparatus for determining a non-apparent attribute of an object having a sensor portion with which the object makes contact and to which the object applies pressure. The apparatus has a computer in communication with the sensor portion that receives signals from the sensor portion corresponding to the contact and pressure applied to the sensor portion, and determines from the signals the non-apparent attribute. The apparatus has an output in communication with the computer that identifies the non-apparent attribute determined by the computer. A method for determining a non-apparent attribute of an object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a nonprovisional of U.S. provisional application Ser. No.61/955,406 filed on Mar. 19, 2014, incorporated by reference herein.

FIELD OF THE INVENTION

The invention involves techniques and a system to acquire and analyzepressure contact patterns. (As used herein, references to the “presentinvention” or “invention” relate to exemplary embodiments and notnecessarily to every embodiment encompassed by the appended claims.)More specifically, the invention provides a system that is able to learnproperties of objects and their behaviors by analyzing data about thepoints of object contact with a surface. Aspects of the inventionutilize pressure data from the forces the objects impart on a pressuresensing surface. The analysis of the data involves one or more layers ofclassification and sequencing. The objects can be inanimate, includingbut not limited to immobile objects and cars, trucks, and forklifts, orhumans or other agents, such as robots, remotely controlled vehicles,etc. Inferred object properties, behaviors, and/or intents can beassociated with objects and permit the system to perform various usefuland beneficial actions. The system is able to determine certain objectstates, including but not limited to detecting when a person is walkingor when they are in an awkward pose. The invention enables automateddetection and tracking of path and activity, and inference of behaviorsand other states of the agent based on the path and features of theirrespective pressure patterns. This system is able to learn, over timethrough the use of what are generally referred to as machine learningalgorithms, to identify additional behaviors at progressively morecomplex levels of abstraction. Human or agent intent can be learned fromthe identification of certain behaviors and sequences of behaviorsincluding but not limited to historically gathered and/or generatedresults.

One useful aspect of the invention relates to automatic measurement ofthe properties of a person's gait, balance, general activity, and otherfeatures extracted from the pressure images resulting from the person'sinteraction with pressure measuring sensors on or under the floor anduse of those measurements to predict certain physical capacities of theperson at the time of interaction or at a future date. The inventionprovides a means to monitor changes in gait parameters, physicalcapacities, and activity over time and to report those changes to a useror system locally or in a remote location. The invention provides anautomated and objective procedure that can be substituted for humanjudgments of a person's capacity to maintain balance and their risk offalling. Predictions of diminished physical capacities are used topredict the propensity to fall, and changes in gait parameters canmonitor the progression of certain diseases. The measurements of gait,balance, and activity can be used to monitor the person's ergonomicperformance and provide a means to give feedback information to theworker or to a system in order to reduce risk or increase the overallperformance and efficiency of the work and the system.

Another useful aspect of the invention relates to identifying propertiesof objects, people and agents on a surface and tracking them over time.These properties and the locomotion paths are used to infer variousactivities and behaviors of the objects, including but not limited theattribution of mental or affective states and intentions. Theseinferences are useful, for example in settings of general activitymonitoring, security systems, and commercial retailing. The inventioncan provide information to other systems or act on the information tomonitor, make a decision, or take an action. For example, the systemmight make a commercial offer to a person or group of people who areshopping, or it might flag the activity of a person or object asunusual, suspicious, or potentially nefarious.

BACKGROUND OF THE INVENTION

This section is intended to introduce the reader to various aspects ofthe art that may be related to various aspects of the present invention.The following discussion is intended to provide information tofacilitate a better understanding of the present invention. Accordingly,it should be understood that statements in the following discussion areto be read in this light, and not as admissions of prior art.

The problem of detection of a human or agent's properties, behaviors,and intents has a long history. Much of the work has involved theacquisition of data by non-contact means. Most commonly, visual imageshave been acquired from various types of cameras and those images havebeen processed to infer salient information. While most non-contact dataacquisition has been in the visible or near-visible part of thespectrum, other non-contact techniques to acquire observations have beenused, including ones based on acoustic and infrared ranging sensing.

Use of data from direct contact with a sensing surface has also beenexploited. These include occlusion systems, where the object is sensedbecause it blocks electromagnetic radiation in parts of the spectrum.There has also been prior work on extracting features from floor surfaceimage data, including identification of objects (U.S. Pat. No.8,138,882), imaging of footsteps and parts of people, and from localsurface pressure measurements to characterize gait (U.S. Pat. No.5,952,585), and to identify people. There does also exist work ontracking people using floor-sensed data.

This direct contact sensing work has used several sensing techniques.Branzel et al. (2013) describe a system that uses a camera mountedbeneath a floor to image the contact of people and other objects withthe floor. The system processes the visual images of the pressurecontact of the objects rather than measurements of the pressure impartedby the object. Connected component analysis is applied to the imagepixels to identify and track continuous areas of contact. Imagefeatures, such as image moments and shape descriptors were calculatedand used as inputs to a trained feed forward neural network to assign aprobability the image area matched one of seven object labels, e.g.hand, shoe, previously stored in a database. A set of heuristicassociation rules are used to identify particular locationconfigurations of identified pressure image parts as being one of fiveposes: standing, kneeling, sitting on the floor, sitting on cube seat orsofa, and lying on a sofa.

Person tracking with floor-sensed data from pressure-activated switchesor capacitive sensors Bibliography entry uses only the successiveactivation of the on-off pressure sensors for tracking by sensingcontact. These systems do use other features of a person's locomotion,including gait parameters, or distinguish features of the path, such asturning characteristics, or actions along the path. Pressure is notsensed directly, although certain events, such as something striking thesurface can be inferred.

Bibliography entry shows how sensed movement trajectories using acapacitive sensing system can be combined with heuristic rules todistinguish ‘normal’ behavior from unusual trajectories. For example,footsteps that begin at a window entrance may be indicative of abreak-in in a home with a floor sensing system.

U.S. Pat. No. 8,138,882 discloses a system to identify an object on afloor by matching the detected contact shape with a shape profile storedin a database and then performing an action if some threshold values areexceeded. It describes applications in securing a premises, for example,detecting whether a person is authorized to be in a location, or that achild has entered the premises. The system can then take some action,for example, to alter lighting or notify the police or a caregiver.

U.S. Pat. No. 5,952,585 discloses a pressure sensing array apparatusthat uses a plurality of current driven electrode pressure sensors tomeasure properties of footsteps and gait. Commercial versions, e.g. theGaitrite system by CIR Systems, are used by clinicians to measure thegait parameters of a single person in a defined protocol of walking tosupport physical therapy after injuries, make analysis of gait toimprove athletic performance, and to monitor the progression of certaindiseases.

Although there has been work using surface contact images and pressuredata to identify objects, parts of people, animals and agents, it isvaluable to be able to infer specifically the classification, actions,behaviors, intents, and other properties of humans and agents in ageneral way and automatically. In the case of humans, such otherproperties include age range, gender, whether or not they have somelevel of knowledge, whether they are in a state of making a decision,whether they are searching for an object or information, whether theyare having difficulty performing a task, and so on. Previous work on useof contact images and localized pressure data has focused only on theextraction of physical properties and their association with features inthe pressure profile, or over time, for example in tracking physicallocation. This invention relates to extracting both physical propertiesof objects and associating mental and other states and properties of ahuman or agent, including classifications such as activity states,gender, and age, which are not immediately discernible in the pressureimage and collections of pressure images.

Gait and footsteps have been studied to extract behavioral biometricsBibliography entry. Gait has usually been studied via visual recognitionsystems and investigated for use in fields such as surveillance, medicalapplications, design and selection of sports shoes, and analysis ofathletic motions and performance.

One specific and valuable problem is to measure the changing physicaland mental capacities of the elderly. Heretofore, this has beenperformed mostly by expert human judgment based on interviewing theperson, making physical measurements, and assessing the results ofstandard physical or mental performance tests. Such assessments can beused to assist the person to anticipate deteriorating abilities, adjustactivities, and otherwise ameliorate the person's situation.

An acute problem is the prediction of a propensity to fall. Fallsamongst the elderly are a significant cause of morbidity and mortality.Age-related reduction of physical capacities expressed in posturalbalance and gait function have been consistently linked with falls.Therefore, various laboratory and clinical tests of balance and gaithave been used in attempts to predict the risk of falling.

Current practice is for a professional to conduct several tests of theperson's balance and gait in a controlled setting, usually a doctor'soffice or clinic. Typical tests include timing the patient to rise froma seated position and begin walking (TUG test), and making gaitmeasurements using systems, such as one available from Zenometrics LLC.Using this data, the expert makes a judgment about the person's physicalcapacity and their propensity to fall. One problem is that thesejudgments are somewhat subjective and experts may vary in their judgmentgiven the same data and/or observations. Further, because the testingprocedure takes place in a clinical setting, there are significantimpediments to monitoring a patient as their physical capacities declinewith age. These include cost of each visit and test, the availability ofexperts, and the continuity, availability, and consistency of recordsover time. Further, there are benefits to making gait observationsduring a person's natural activities because research shows that gaitparameters measured in clinical settings are significantly differentthan those measured when a person is outside the clinical settingBibliography entry. In addition, there is benefit to frequentobservations of a person's gait.

It is desirable to have an automated procedure that predicts thepropensity to fall because it can be used constantly and because it canbe used by non-experts, including the elderly person and those with theresponsibility of care giving. Another important benefit of an automatedprocedure is its ability to facilitate objective decision-making aboutfuture risks that can result in more optimal cost-benefit choices aboutthe elderly person's living situation, for example, whether continuedindependent living is significantly more risky than before.

The problem of making an objective automated solution requires a meansto automatically collect the observations and providing a computablerepresentation that can be used to produce automatic objective judgmentsof the risk of falling that correlate well with human-based assessments.To achieve broad application of the solution in non-clinical settings itis desirable to be able to collect observations and analyze normalwalking activities.

Gait characteristics have been correlated with human cognitivecapacities and states, both in terms of current activities and futurecapacities. Certain pathological conditions are believed to affect brainfunction in ways that affect gait, for example by impacting motorcontrol or split attention capacities Bibliography entry. Gait analysisand physiological tests such as rising from a seated position andbeginning to walk (TUG test) can also be used to detect and predictcognitive declines of an individual. Greene describes such a system inEP2688006 A2 where inertial sensors attached to a person are used todetect gait characteristics and the inertial data can be used by aclassifier to predict the future cognitive capacity of the person basedon changes in the inertial data compared to a baseline.

One cause of work-related musculoskeletal disorder injuries for assemblyline workers is repetitive actions where the worker assumes an awkwardpose. Such injuries are estimated to cost many billions of dollarsannually in compensation costs, lost wages, and lost production. Theautomotive industry has been a leader in this area because ofsignificant costs due to injuries, legal action by unions, state workercompensation boards, and the insurance industry. Recognized benefits ofgood ergonomic design include increased factory efficiency and productquality.

Pressure patterns have been used to identify people Bibliography entryusing the trajectory of the center of pressure in gait and pressurepatterns of the individual footsteps. While high identification ratescan be achieved in controlled settings, it is desirable to improveidentification rates. Additional independent features from pressuremeasurements at different time scales, for example, gait parameters andlocomotion behaviors, can improve classification and identificationrates and reduce false positives.

Locomotion is under cognitive control and velocity patterns allowinference of locomotion goals, for example people orient themselves inways that reflect their visual attention Bibliography entry. Visual datahas been analyzed and used to automatically classify individual activityand behavior, for example as shown by Bibliography entry, toautomatically identify anomalous behavior, for example as disclosed inU.S. Pat. No. 8,494,222, and to identify social relationships betweenpeople, for example as disclosed in U.S. Pat. No. 7,953,690.

Visual image-based systems have various challenges and limitations,including acquisition and simultaneous tracking of multiple objects atdiffering distances and with changing scale. Objects can be partially orfully-occluded and salient visual features may be cloaked. These typesof limitations contribute to uncertainty in recognition andidentification of objects and their properties, as well as touncertainties in any further analysis of the data or taking action basedon the data. Such uncertainties and other difficulties limit the utilityof such systems in many settings, including automated securityapplications. They also make it difficult to extend such systems toprovide new types of utility, for example attribution of behaviors topeople that can be interpreted in specific contexts, such as retailshopping activity and intent. Useful object properties such as weightand detailed gait and locomotion parameters are hard to extract inuncontrolled settings. Visual systems also tend to have demandingprocessing requirements for large scale multi-object settings because ofthe need to process the images to perform basic object recognition anddiscern other structure in the image.

The invention makes classifications of objects and inferences about theproperties and behaviors of objects based on direct measurement of theobject's contact with a surface. In this way the invention solves manyof the challenges and limitations of applying established methodologiesand inference techniques to visual data in order to make objectclassifications and inferences with higher quality and better systemperformance.

OBJECTS OF THE INVENTION

It is an object of this invention to provide a means to automaticallycollect contact and/or pressure measurements of contact and locomotionof one or more objects on a surface.

Another object of this invention is to collect such measurements overtime and make calculations on those measurements, especially changes inthe values.

Another object of the invention is to infer properties of an object oragent that are not directly observable in contact or pressuremeasurements, for example as general classification of the object or ahuman in groupings of age, mental or physical capacity, or gender, agentactions or behaviors or intents, etc.

Another object of the invention is to track objects including humans andagents on a surface.

Another object of the invention is to differentiate and/or identifypeople moving on a surface.

Another object of this invention is to provide a means to record andmonitor the path of an object or agent from patterns of contact and/orpressure on a floor surface.

Yet another object of the invention is to measure aspects of gait, thegait velocity and balance of a person while they are standing, walkingor otherwise moving.

Another object of the invention is to calculate the future risk offalling from measurements and/or repeated measurements of the gaitvelocity and balance of a person.

Another object of the invention is to calculate the future cognitivecapacity from measurements and/or repeated measurements of the gaitvelocity and balance of a person.

Yet another object of the invention is to identify patterns of thechanges in measurements and to calculate the correlation of thosepatterns with changes in patterns of expert judgments about the futurerisk of falling from a reference collection.

Yet another object of the invention is to identify patterns of thechanges in measurements and to calculate the correlation of thosepatterns with changes in patterns of expert judgments about the futurecognitive capacity from a reference collection.

Another object of this invention is to provide a means to infer theagent activity state and activity sequences from patterns of contactand/or pressure on a floor surface.

Another object is to record worker actions in a work environment,including detection of awkward poses, to support an ergonomic evaluationof the worker's activity in the work environment.

Another object is to measure the ergonomic parameters of a worker'sactivity to provide feedback to the worker and/or a system to modify theenvironment to improve the worker's ergonomic situation.

Another object of the invention is to provide a means to attributebehaviors to one or more objects, both individually and collectively.

Another object of the invention is to provide a means to infer socialand other relationships between an agent and one or more other agents.

Another object of the invention is to provide a means to attributeintention to an agent based on patterns of contact and/or pressure on afloor surface.

Yet another object of the invention is to use inferences from various ofthe invention objects as evidence for systems that communicate or signalthe agent for some purpose, for example by making a commercial offer,providing information, making a warning, or changing some aspect of theenvironment, such as lighting.

Yet another object of the invention is to use inferences from various ofthe invention objects as evidence for systems that make decisions aboutthe agent for some purpose, for example by classifying the agent or theagent's activities or behavior.

Yet another object of the invention is to provide a means to provideinformation to another system about the properties, actions, behaviorsor intents of the agent(s), for example by tracking, classifying oridentifying the agent(s) or their actions, behaviors and/or intents. Aspecific example is identifying unusual or threatening behavior.

An additional object of the invention is to provide a technique forinferring mental and/or internal states of an agent without requiring apriori knowledge of the agent properties, the tasks in which the agentis engaged, or of the agent's intentions, or the contents or locationsof specific regions that situate the agent.

Another object of the invention is to simultaneously perform one or moreof the above invention objects for multiple humans and/or agents on aunified or distributed contact or pressure sensing surface.

Still other objects and advantages of the invention will in part beobvious and will in part be apparent from the specification anddrawings.

The utility and advantages of the ability to accomplish the aboveinvention objects includes the ability to automate such inferences andpredictions and take appropriate action, including to assist, inform, orotherwise interact with the object or agent. Examples include automaticuser interface changes, adjustment of machinery, tools or other tangibleobjects in order to better accomplish a task, identification of acontext for the agent in order to decide to provide appropriateinformation, engage in a dialog, making an offer, etc. Another generalbenefit of achieving one or more of the invention objects is to use theinformation about the agent and the agent's context in other systems,for example aggregating audiences for a product brand, which is ofsignificant value to advertisers, or using the information to makeindividual models of the user or general user models that are valuablein other contexts, for example to provide support, etc. in future tasks,such as information search or emergency intervention.

BRIEF SUMMARY OF THE INVENTION

These objects and advantages are provided by a computer-implementedmethod for inferring but not limited to the following: physical,behavioral, mental and internal states of an agent, and other propertiesof an object or agent, from measurements of contact or pressure impartedby the agent onto a pressure-sensing surface. The method includesidentifying contact points and/or elementary features of thespatio-temporal pressure data, such as contact positions and force, andchanges thereof.

The use of measurements of contact and pressure provide directmeasurements of objects allow the invention to overcome many limitationsand practical implementation problems that arise because of thechallenges of processing remotely sensed data in image-based systems.Many representation, recognition and inference techniques used forvisual data can be applied to contact and pressure data with suitableadjustments that will be apparent to those practiced in the art of dataanalysis and machine learning. Usable representations of an object canbe fashioned from contact and/or pressure measurements using far fewerdata samples than are required to process visual images. This permitsthe invention to better overcome certain data processing and storagechallenges that can arise in multi-object settings and for analysis ofdata from large areas and for situations where the analysis resourceoverhead needs to be minimized, for example to provide fast analysis andoutput to an application that acts on the analysis result.

Sensing apparatuses that can make inertial measurements of movement andactivity attached directly to the objects have practical limitationswhen it is desirable to learn properties of objects on a surface in anunobtrusive manner and over extended periods of time. The use ofmeasurements of contact and pressure gathered from surface contact hasadvantages. All of the objects on a surface generate contact andpressure data, not just those specially instrumented, and themeasurements have a uniform scale with no special calibration orvalidation required to enable comparison between object contact andpressure properties or for derived features and properties. Theseadvantages are apparent and confer utility for sensed surfaces inuncontrolled environments, and when a plurality of objects need to beprocessed, and when the data is collected or compared over extendedperiods of time.

In order to automatically collect data about locomotion across a surfacea contact and/or pressure sensing system is used. In the preferredembodiment, it employs a plurality of arrayed pressure sensing elementsfor example by using a mechanically interpolating pressure sensingsystem such as Tactonic Technology LLC's pressure sensing floor tiles.The system provides direct measurement of the contact and/or pressureimparted by an object on the floor for each unit of time.

In order to overcome the basic problem of inferring properties ofobjects, agents, and humans that are not immediately discernible fromcontact or pressure data, data is collected from a surface contact orpressure sensing system and that data is analyzed as a time series or inother ways. The analysis can be on-line or deferred for laterprocessing. The analysis consists of a combination of signal processingand machine learning using the surface contact or pressure data, andvarious derivations thereof, Including but not limited to a time varyingimages of the applied pressures. The machine learning techniques includeunsupervised, semi-supervised, and supervised methods known to thosepracticed in the art of mathematical modeling.

The collection of pressure data observations is processed to makenumerical representations of the contact and/or pressure at the sensorpoints from digital or analog pressure values. The numericalrepresentations are then processed using well-known clusteringtechniques, for example by using density-based clustering, to identifyregions that have contact or pressure measurements of a distinct object,for example a human foot or a wheel. These measurements of the objectsare then analyzed using statistical and other techniques known to thoseskilled in the art to make mathematical representations of the objectson the surface at a time and over time segments as appropriate to thedesired result. Learned patterns of those representations can beclassified using models derived from processing data for known examplesof each class, or as a result of heuristic rules from human experienceor association rules learned from data sets or learned Markov LogicNetworks. Unsupervised techniques, such as but not limited to clusteringand association rule mining, may also be used.

The data to process can be drawn from differing time scales orcollection sampling as appropriate to the classification and predictivegoal of the analysis and other requirements such as for on-line analysisor real-time prediction. For example, to measure human foot stepproperties, including balance, it is desirable to have a collection ofmoment to moment pressure measurements over the foot contact area. Tomeasure gait parameters, such as velocity, it is desirable to havemeasurements of successive footsteps.

From pressure measurements and patterns of pressure measurements,properties of individual objects moving on a contiguous or distributedpressure surface can be measured and inferred. These properties can bedirect physical properties of the object, such as its weight, velocity,and center of mass. Pressure measurements and pressure patterns overtime provide a sufficient number of features to distinguish and identifyobjects, including people, using machine learning techniques includingbut not limited to clustering, decision trees, and other modelingtechniques. Such techniques may be used alone or in combination withother modeling techniques to achieve useful prediction performance.

Use of statistical techniques to compare of patterns of gait and/orbalance over time for an individual can indicate physical deteriorationand can be projected to allow prediction of the probability of fallingin some future time period.

The patterns of pressure measurements and derived object features inlocomotion can be analyzed as a sequence of coherent path segments.These path segments can be classified using supervised, semi-supervised,or unsupervised machine learning techniques to infer the probablebehavior of the agent. For example, an agent's pattern of path segmentsin a retail space, such as a mall, can distinguish different behaviors,for example browsing vs. shopping with a specific purchase goal.Further, such learned classification of changes in activity andlocomotion states, for example orienting to a new direction and thenslowing down or stopping, can be indicative of agent mental states,including information acquisition and decision making. Such observationsof behaviors can be classified using models or by making similaritycomparisons with stored behavior instances or general models ofbehaviors. Unusual or novel behaviors can be detected by settingappropriate similarity thresholds or applying heuristic or learneddecision procedures. Analysis and association of the path segments ofone agent with the path segments of other agents can indicate social orother relationships between agents, for example people working orshopping together. The confidence of such inferences can be improvedwith knowledge about location and other context features added to theinput for modeling, for example the location of a store entrance or adeparture gate in an airport terminal can aid human or automatedrule-based interpretation of path-segment and activity sequencebehaviors.

The invention accordingly comprises the several steps and the relationof one or more of such steps with respect to each of the others, and theapparatus embodying features of construction, combinations of elementsand arrangement of parts that are adapted to affect such steps, all isexemplified in the following detailed disclosure, and the scope of theinvention will be indicated in the claims.

The present invention pertains to an apparatus for determining anon-apparent attribute of an object. The apparatus comprises a sensorportion with which the object makes contact and to which the objectapplies pressure. The apparatus comprises a computer in communicationwith the sensor portion that receives signals from the sensor portioncorresponding to the contact and pressure applied to the sensor portion,and determines from the signals the non-apparent attribute. Theapparatus comprises an output in communication with the computer thatidentifies the non-apparent attribute determined by the computer.

The present invention pertains to a method for determining anon-apparent attribute of an object. The method comprises the steps ofmaking contact and applying pressure with an object to a sensor portion.There is the step of receiving signals by a computer in communicationwith the sensor portion from the sensor portion corresponding to thecontact and pressure applied to the sensor portion. There is the step ofdetermining by the computer from the signals the non-apparent attribute.There is the step of identifying at an output in communication with thecomputer the non-apparent attribute determined by the computer.

The present invention pertains to a computer-implemented method to learnclassifications and properties of one or more objects by processing asequence of surface contact and/or pressure measurements captured by aplurality of local contact or pressure sensing system at a time and overtime. The method comprises the steps of receiving a frame or frames ofthe sequence which includes data for a plurality of contact or pressuremeasurements included in the frame. There is the step of identifying oneor more collections of contact and/or pressure measurements in theframe, where each collection represents an object on the surface. Thereis the step of generating models to extract one or more features fromthe contact and/or pressure measurement collections that are associatedwith each identified object. There is the step of extracting a pluralityof features from the collections of contact and/or pressuremeasurements. There is the step of classifying each of the collectionsof contact and/or pressure measurements using a trained or untrainedclassifier. There is the step of supplying the extracted features and/orthe object classifications from one or more frames to a machine learningengine. There is the step of using the machine learning engine togenerate values for one or more properties of one or more objects and/orgenerate semantic representations of the behavior of one or more objectsover a plurality of frames, where the machine learning engine isconfigured to learn properties and behavior patterns observed in thecontact and/or surface pressure measurements over the plurality offrames and to identify patterns of behavior by the classified objects.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

In the accompanying drawings, the preferred embodiment of the inventionand preferred methods of practicing the invention are illustrated inwhich:

FIG. 1 shows the several levels of acquiring and processing the surfacedata observations.

FIG. 2 shows a relationship of a machine learning module to the computersystem.

FIG. 3 shows an example Markov state model of an object.

FIG. 4 shows a method of processing surface data observations to infersocial relationships between agents on the surface.

FIG. 5 shows a specific system to infer the risk of falling.

FIG. 6 shows a specific system to infer social relationships between oneor more people.

FIG. 7 shows a foot pressure data pattern where the heel of a left foothas just made contact with the sensing surface.

FIG. 8 shows a foot pressure data pattern where the left foot is in fullcontact with the sensing surface and the right foot is in the swingphase and does not have contact.

FIG. 9 shows a foot pressure data pattern where both feet are in contactwhile the transfer to the right foot progresses.

FIG. 10 shows a foot pressure data pattern where the left foot isleaving the surface.

FIG. 11 shows a foot pressure data pattern where the right foot is infull contact with the surface and the left foot is in the swing phaseand does not have contact.

DETAILED DESCRIPTION OF THE INVENTION

Referring now to the drawings wherein like reference numerals refer tosimilar or identical parts throughout the several views, and morespecifically to FIGS. 2 and 5 thereof, there is shown an apparatus 100for determining a non-apparent attribute of an object 41. The apparatus100 comprises a sensor portion 21 with which the object 41 makes contactand to which the object 41 applies pressure. The apparatus 100 comprisesa computer 19 in communication with the sensor portion 21 that receivessignals from the sensor portion 21 corresponding to the contact andpressure applied to the sensor portion 21, and determines from thesignals the non-apparent attribute. The apparatus 100 comprises anoutput 11 in communication with the computer 19 that identifies thenon-apparent attribute determined by the computer 19.

The object 41 may be a person, the non-apparent attribute may be acognitive capacity decline, the sensor portion 21 may be a plurality ofsensor tiles forming a walkway, and the signals may be indicative ofgait of the person moving on the walkway.

The present invention pertains to a method for determining anon-apparent attribute of an object 41. The method comprises the stepsof making contact and applying pressure with an object 41 to a sensorportion 21. There is the step of receiving signals by a computer 19 incommunication with the sensor portion 21 from the sensor portion 21corresponding to the contact and pressure applied to the sensor portion21. There is the step of determining by the computer 19 from the signalsthe non-apparent attribute. There is the step of identifying at anoutput 11 in communication with the computer 19 the non-apparentattribute determined by the computer 19.

The object 41 may be a person, the non-apparent attribute may be acognitive capacity decline, the sensor portion 21 may be a plurality ofsensor tiles forming a walkway, and the signals may be indicative ofgait of the person moving on the walkway.

The present invention pertains to a computer-implemented method to learnclassifications and properties of one or more objects 41 by processing asequence of surface contact and/or pressure measurements captured by aplurality of local contact or pressure sensing system at a time and overtime. The method comprises the steps of receiving a frame or frames ofthe sequence which includes data for a plurality of contact or pressuremeasurements included in the frame. There is the step of identifying oneor more collections of contact and/or pressure measurements in theframe, where each collection represents an object 41 on the surface.There is the step of generating models to extract one or more featuresfrom the contact and/or pressure measurement collections that areassociated with each identified object 41. There is the step ofextracting a plurality of features from the collections of contactand/or pressure measurements. There is the step of classifying each ofthe collections of contact and/or pressure measurements using a trainedor untrained classifier. There is the step of supplying the extractedfeatures and/or the object 41 classifications from one or more frames toa machine learning engine. There is the step of using the machinelearning engine to generate values for one or more properties of one ormore objects 41 and/or generate semantic representations of the behaviorof one or more objects 41 over a plurality of frames, where the machinelearning engine is configured to learn properties and behavior patternsobserved in the contact and/or surface pressure measurements over theplurality of frames and to identify patterns of behavior by theclassified objects 41.

There may be the steps of locating one or more objects 41 on the surfaceby detecting one or more features using a plurality of featureprediction models applied to the entire surface or to regularly orrandomly selected regions of the surface, or by using similarity to anexemplar or to a reference collection of examples or to previouslyobserved objects 41, applied to the entire surface or to regularly orrandomly selected regions of the surface. There may be the stepslocating and tracking one or more objects 41 on the surface by detectingtheir current position in a space-time window and calculating theoverlap with previous detected positions or by predicting the nextposition of the object 41 or objects 41 using learned models from theconfigured machine learning engine or by applying a search algorithmusing the previously detected positions.

There may be the step of reacquisition of one or more objects 41 withsome level of confidence when they reenter the surface by comparingdirectly or with some feature transform their feature signature withstored models of objects 41 accessible by the system. There may be thestep of recording the location and monitoring the paths of one or moreobjects 41 on the surface stored in local memory or remotely. There maybe the step of the segmentation of the recorded paths of one or moreobjects 41 into units using learned models from the configured machinelearning engine or by applying heuristic rules or rules learned fromobservations not based on contact or pressure data. There may be thestep of the segmentation of the recorded paths of one or more objects 41into units using measurements from other systems in real time or fromstored data of the same objects 41 or exemplars of the objects 41. Theremay be the step of configuring of the machine learning module to inferproperties of objects 41 that are correlated with contact and pressureproperties of objects 41. There may be the step of identifying peoplefrom combinations of inferred properties and contact and/or pressurepatterns.

There may be the step of identifying people from combinations ofinferred properties and contact and/or pressure patterns and theirlocation. There may be the step of calculating gait velocity and balanceusing learned models from the configured machine learning engine. Theremay be the step of using changes in patterns of gait velocity andbalance measurements recorded at disparate times to calculate the futurerisk of falling using learned models from the configured machinelearning engine. There may be the step of using changes in patterns ofgait velocity and balance measurements recorded at disparate times andcorrelating them with changes in a reference collection of expertjudgments about the future risk of falling based on sequences of gaitvelocity and balance measurements recorded at disparate times.

There may be the step of using learned models from the configuredmachine learning engine to predict the activity state and activitysequences of an agent. There may be the step of using observed statemodels and heuristically-derived state models to predict the activitystate and activity sequences of an agent. There may be the steps ofrecording and monitoring the actions of a worker and evaluatingergonomic performance. There may be the step of providing ergonomicstate and performance information to a worker, a manufacturing system,or a local or remote information system and record and store theinformation locally or remotely for later use or as an input to anothersystem to make decisions, task action, or otherwise process theinformation. There may be the step of using learned models from theconfigured machine learning engine to predict the mental states ofagents.

There may be the step of using learned models from the configuredmachine learning engine to attribute an intention to an agent. There maybe the step of inferring social relationships between objects 41. Theremay be the step of taking actions, signaling an agent, or makingdecisions. There may be the step of providing information to anothersystem as evidence to take action, signal an agent, or make decisions.There may be the step of carrying out on one or more combinations ofobjects 41 on the surface. There may be the step of using changes inpatterns of gait velocity and balance measurements recorded at disparatetimes to calculate the rate of cognitive capacity decline using learnedmodels from the configured machine learning engine.

There may be the step of using changes in patterns of gait velocity andbalance measurements recorded at disparate times and correlating themwith changes in a reference collection of expert judgments about therate of cognitive capacity decline based on sequences of gait velocityand balance measurements recorded at disparate times.

In the operation of the invention, the basic embodiment of the inventionconsists of a system to detect a plurality of localized contactmeasurements with a surface, a system for collecting the measurementsfrom one or more sensing units, and a sequence of analysis steps to makeone or more representations of an object's surface contact interaction.In a preferred embodiment of the invention the system, detectinglocalized contact with a surface, consists of a plurality of pressuresensing elements in an array on a surface and a system for collectingthe measurements from each of the pressure-sensing units. The contactsensing surface may cover a floor in part or in whole or is integratedwith or lays under a floor, but the localized contact detection systemor pressure sensing elements might be deployed to cover or be integratedinto or placed under, in part or in whole, various surfaces. Suchsurfaces include, but are not limited to, walls, tables, furniture,decks in vehicles and ships, uneven or multilevel floors, sports andtraining surfaces, roads, or land forms. The sensing units may beintegrated into a pressure sensing surface or otherwise cover, beintegrated with, or be beneath regions of the surface. A specificembodiment of the invention uses pressure sensing arrays such as themechanically interpolating pressure sensing tiles provided by TactonicTechnologies LLC.

The localized contact detection system or pressure sensing surface maybe deployed on a permanent, semi-permanent or temporary basis and maynot necessarily be restricted by environment. For example the surfacecan be in a doctor's office or a clinic, indoors or outdoors, in apublic or private space, or in a factory or a residence, or integratedwith a road or other paved surface.

The employed sensing system can monitor substantially all of the surfaceor just a limited portion thereof. The portion monitored can becontiguous or as separated patches or a collection of points. As theobject, such as person or agent, makes contact or moves on the surface,signals are generated by one or more of the sensor units. These signalsare collected by electronic systems and converted to a digital signal.Such a digital representation is convenient for the data processingsteps, but it can be seen that a digital signal is not required for theprocessing steps, as any mathematical representation of the array ofcontact or pressure signals is sufficient as an input to the processing.

In one specific embodiment, a one or more pressure sensing tiles, eachwith a plurality of pressure sensing units or elements, may be deployedto cover a run in a residential hallway. Selection of a location mightbe made to have cases where a person can be expected to walk a number oftimes during the day, or in a location where routine or expectedbehaviors might be inferred, for example use of a bathroom or entry/exitfrom a bedroom. In this way, in addition to the direct pressuremeasurement of the agent/person, the timing and frequency ofobservations may also provide information for extraction of features orinferring the behavior, intent, or changes in personal habits of theagent/person. In a general way such types of context information can beunderstood to be of use for analysis and interpretation of contactand/or pressure measurement data in various concrete embodiments of thesystem. The descriptions of the analysis used in the invention should beunderstood to be extended to use such information appropriately, both asfeatures to learn models and as inputs to models used for calculation,and in rules relating to the creation and use of models and inferences.Such uses of the models and inferences by the invention can includecommunication with humans or other systems and decisions to applyapplication programs or systems.

After collecting information from the sensor elements over any finiteunit of time, the electronic processing unit for the contact or pressuresensing tiles will record the data. That data can be stored for off-lineanalysis or streamed to a system for continuous analysis.

In the descriptions of invention embodiments, it is to be understoodthat object, agent, and human or person are to be used interchangeablywhere it is reasonable to do so. For example, general feature extractioncan relate to any property of an object, including agents and humans. Onthe other hand, when attributing intent or mental states it is to beunderstood that only objects capable of having such properties areintended as the referents. References to objects, agents, persons andhuman are not intended to be limiting in any way in the following and donot exclude animals, mechanical devices or systems, or composites ofobjects. For example, it is to be understood that the inference of abehavior or intent for a controlled vehicle is within the scope of theinvention.

FIG. 1 provides a general sketch and illustration of the data processinglevels in the invention. It can be seen that the pressure patternrecognition and interpretation system 12 consists of multiple processinglevels that provide progressively higher-level or more abstractrepresentations of objects on a surface. The operations illustrated inFIG. 1 generally consist of accepting a set of data inputs 1 directlyfrom the contact and/or pressure sensing system or from a system thatstores the data from the contact and/or pressure sensing system. Theinput data can be processed as complete frames covering an area or timesegment, or as samples of the input data, for example as a collection ofregions on the surface or a time segment sample of the sensor data or adata sample based on some other dimension. At LEVEL 0 an array ofpressure data values 2, which can be pressure measurements or localcontact indications categorically encoded to correspond to ‘contact’ or‘no contact’, is created from the input. LEVEL 1 processes the LEVEL 0information to determine basic features of the observations 3 and makesa pressure or contact representation. LEVEL 2 analyses the contactand/or pressure representation to extract patterns and features of therepresentation 4. Such features can include object recognition andobject properties. LEVEL 3 processes the LEVEL 2 results to learn objectmovement patterns 5. LEVEL 4 and above infers object movement behaviorand intent 6. The objects and their behavior are analyzed using avariety of statistical modeling techniques applied to the input data andto representation results derived from one or more layers of processing.It is to be understood that the results from one level can be used toenrich or improve the results derived at another level, as indicated forexample by 8. This derivation and use of results from any particularlevel is to be understood as applying in a general way and the resultsof processing at any level provides features and interpretation rulesthat can provide inputs or decision criteria useful for processing atany other layer or a plurality of layers and may be combined withoutlimitation. For example 7 might indicate a classification model derivedat LEVEL 3 5 being used to impute missing data in LEVEL 0 2. Generally,the results of the pressure pattern recognition and interpretationsystem 12 can be made available to one or more application programs 11.It is to be understood that the results can consist of results from onelevel of processing or of any mixture of levels of processing and theycan be individually and collectively used as inputs to applicationprograms in local or remote systems that achieve particular utility. Forexample, applications that present predictions of the probability of aperson falling in some forecast time period, identification ofauthorized or unauthorized persons or objects in a security zone, takingnotice of or acting on anomalous object or agent behaviors, orpredicting properties of people, such as their state of attention ordecision making, or social relationships with other people. The resultscan be stored locally or remotely for analysis or later use. Further,the results and/or interpretation of the results of the inventionprocessing can be communicated to a human or some other system, or canbe used in a decision-making process that might invoke an action,including selection and execution of an application program orcommunication with some system. Particular embodiments of the inventionprovide details of the processing at various levels, especially as theyrelate to achieving objects of the invention and providing specificutility.

Another aspect of the embodiment of the invention has acomputer-implemented system as illustrated in FIG. 2 which includes asystem having the contact and/or pressure sensing input source 21configured to provide a sequence of contact and/or pressure imageframes, each depicting the contact and/or pressure measurements on thesurface at a time, a processor 14, which may include a hardwareprocessor 14, and memory 18, which may include a non-transitory storagemedium, containing modules and programs to process the input contactand/or pressure data 16, 18. When executed on the processor the contactand/or pressure data processing module program 16 provides a suitablerepresentation of the data that can be output 13 or used as an input tothe modeling module 17. When executed on the processor the machinelearning program 17 performs operations that analyze the contact and/orpressure measurements at a time and over a sequence of such pressureimage frames to carry out one or more of the steps in FIG. 1.

In the preferred embodiment, one or multiple objects on the surface arelocated by processing the input data or by sampling the input data tofind positive contact and/or pressure measurements. The input data maybe sampled using a regular sampling pattern or a random samplingprocedure, with or without constraints. When a positive result isdetected nearby sensor outputs are processed to learn the boundaries andcontact and/or pressure profile of the object.

One embodiment of the invention uses one or more Hough filters or Houghforests from one or more of the processing levels trained using labeledinput data to assign a probability that a specific type of object or anobject in a particular state or with certain properties is at a locationon the surface based on the contact and/or pressure profile and otherproperties of the data. Any object detection algorithm could be usedhowever, for example detection using density clustering, and thosetrained in the art recognize that many supervised and unsupervisedmachine learning techniques and signal processing techniques can be usedto determine the positions and contact shapes of the objects on thesurface with some level of confidence. A specific object detectionprocess using Hough forests employs the well-known Hough transform andthe random forest technique, an ensemble of random decision trees. Houghforests allow fast application to input data and are also efficientlytrained, and are suitable for interactive applications as well asinvention embodiments that cover large surfaces with many objects on thesurface. A Hough forest can be trained using example contact andpressure data. For each object a bounding area is identified and thearea is associated with the object class and the properties of theobject. For example, an object class might be ‘human’ and a list ofproperties could include ‘female’, ‘age between 20 and 30’, ‘walkingwith friend’, ‘wearing sneakers’ and so on. There is no limitation onthe number or type of properties and they can include relatively stablephysical properties, transient physical properties including motions,and non-physical properties such as intentional states. The selection ofan object bounding area and the associations of class and properties canbe performed manually or using some automated technique, including theapplication of algorithms to find similar examples in databases or inreference systems. For each object, the contact and pressuremeasurements in the bounding area are associated with a vector notingthe absence or presence of each of the properties that are used todetect the objects. The collection of such training examples is theninput to a random forest algorithm to make a classification model thatis used for detection. It is typical that a collection of models arecreated, each for the detection of a single feature. In such cases, thetraining sets note only the presence or absence of the property for theclassification model. The detection of an object proceeds by inputtingmeasurements in a region of the surface that can be selected randomly,or sampled using some procedure, and then applying the collection ofdetection model which vote whether the object or property they detect ispresent. Such a vote can be categorical or not depending on whether aclassification model or a regression model was used. It can be seen thatby successive sampling of the surface, objects and properties of objectscan be detected with some confidence level.

Once an object has been located with sufficient confidence, theinvention continues to monitor and sample the region as appropriate toconfirm the existence and properties of the object and to alter theregion to track the positive contact and/or pressure sensor outputs asthe object moves, either by predicting the expected possible locationsof the object or by using a search algorithm. Such object locations,both identified and predicted, and object properties can be usedimmediately or stored and updated in a database or via some otherpersistence mechanism directly or remotely accessible by a processor.The collection of inferred object properties can be used as an objectidentifier and used to reacquire the object if it leaves the sensingsurface and later returns or enters a different sensing surface. Suchreacquisition could take place even at long time scales but with lowerconfidence based on uncertainties associated with one or more featuresof the object, for example if a person's weight has changedsignificantly. The object properties can be related in a model of theobject and inferences about the object identity, or distinguishing theobject from other objects, can be achieved by suitable calculations onthe model as will be appreciated by those trained in the art. Forexample, a woman might be identified with a collection of features whenentering a store, where she purchases a pair of shoes and changes hershoes. Upon re-entry to the sensing surface the system can applyfunctional transforms to the model that correspond to changes inpressure distribution and gait profile for different shoe types. Asimpler example is the case where the person picks up another object andthus increases their weight. In such cases, the invention can make anidentification prediction with some level of confidence, even when thereare temporal gaps in the observations of the object. Such relationshipsbetween model features and other properties of the model can be learnedempirically from observations using various modeling techniques orcalculated using a suitable causal or probabilistic model of the systemthat imposes such changes on the object or be applied from heuristicrules or learned from association rule mining on observations of theobjects, including image-based or other sensing techniques, or fromother data sets.

A specific example of application of the preferred invention embodiment,the input data can be analyzed to find footsteps and extractmeasurements of footstep properties and the gait of the person. Use ofwalkers, canes and other supportive devices can be identified anddistinguished by their relatively consistent shapes and/or pressurepatterns along with a path trace that coheres with the paths of thefootsteps, for example one expects the path of the support devicecontact points to be roughly parallel to those of the footsteps.

One process by which footsteps can be detected is to identify theboundaries of the locations that have positive contact and/or pressuremeasurements using density-based clustering or combining density-basedclustering with a cluster-center and a distance matrix to identifycontact clusters that likely belong to the same foot. The assertion thatone or more contact clusters are part of the same foot can be confirmedby successful prediction of expected locations and pressure and/orcontact features, including shape and pressure distributions. Thisprocedure can be generalized to associate identified contact andpressure areas with a single object having several areas of contact withthe surface. Other unsupervised, semi-supervised or supervised machinelearning techniques can be used to identify the local contact and/orpressure regions. In multi-object cases, coherent patterns of suchmeasurements will indicate at some confidence level the presence of aperson or object at a location. For example, coherent locomotionpatterns of two feet in a walking gait or some other gait can beidentified and distinguished. Likewise, the locomotion of a wheeledvehicle or other object can be identified by comparison to knownpatterns of contact and pressure characteristics of moving objects. Suchcoherent locomotion contact and/or pressure patterns can also be learnedusing various machine learning techniques, for example by using a hiddenMarkov model or a convolutional neural network.

When the feet are identified from sensor data for the region holding afootstep is collected and the location and properties of the pressuredata in the region can be calculated. One such calculation is the centerof the pressure measurements.

Gait parameters are calculated from the timing and location ofsubsequent footsteps. The calculated gait parameters includeinstantaneous and average gait velocity, the swing time of a footstep,stride length, pronation and supination, duration of distinct footcontact phases and differences of the foot parameters, including eventtimes and locations, from footstep to footstep and stride to stride. Theinvention records, classifies, and compares patterns of gait parametersover time periods. Changes in these patterns are classified using atrained classifier to measure current values of properties of a personand predict future values, such as the likelihood of falling.

An embodiment of the invention infers a person's balance from themoment-to-moment variations in the center of pressure within a footstepand over multiple footsteps. One specific technique is to calculate thestandard deviation of the center of pressure resolved intomedial-lateral and anterior-posterior components relative to the footorientation. Other formulas can be applied to make such measurements ofbalance.

For all object properties and features, patterns of statisticalproperties and sequences of the pressure observations can be recordedand stored. Changes in such pattern properties over time can be used asinputs to calculate historical trends and make predictions about thefuture value of properties and features of an object. One example isprediction of the expected gait velocity and balance of a person. Aspecific utility of such predictive capacity of the invention is thecalculation of the future probability of falling, which has beencorrelated with declines in observed gait velocity and deterioratingbalance. In addition, the pattern of changes produced by the inventionprovide inputs to a classifier trained using expert human judgment ofthe future risk of falling to make predictions that supplement orreplace such expert judgments. These human judgments of the future riskof falling can be based on clinical tests to assess fall risk potential,such as but not limited to a test measuring the time needed for a personto rise from a seated position and begin walking (Timed Up and Go orTUG), or observing the ability of the person to maintain their stancewhen disturbed by a light push. Another specific utility of thepredictive capacity of the invention as applied to gait analysis and thepattern of gait property changes is the prediction of cognitive declinefrom gait velocity and other gait properties as compared to a history ofmeasurements. In the same way the pattern of changes provide inputs to aclassifier trained using expert human judgment of future cognitivecapacity to make predictions that supplement or replace such expertjudgments. These human judgments of cognitive decline can be based onclinical tests including the TUG test amongst others.

The contact and/or pressure data, and any or all of the analyticalrepresentations of the data, for example identification of footsteps,locomotion path segments, or inferred states of an agent, can betransmitted in various ways, including digitally, to a remote locationfor further analysis, distribution, or to provide evidence for adecision and/or action by humans or a computer-implemented system.Additionally, the data and any analytical representations of the datacan be used at the site where it was collected to provide evidence for adecision and/or action by humans or a computer-implemented system.

Object behaviors can be learned by applying techniques to first identifythe locomotion paths of an object and then segmenting the path using atrained classifier. Object paths may also be segmented using heuristicrules or rules learned from other types of data, for example visualobservations of object movements and/or object movement patterns. Thepath segments and the dynamic and persistent features of the objectsduring each path segment are input to a trained classified whose outputis a behavior label for the object. A specific embodiment useslocomotion traces and the associated pressure- and non-pressure-basedfeatures to train a Markov model of human movement. FIG. 3 illustratesan example of a generic Markov state model. Each state, for example,STATE_4 27, is associated with a pose, for example standing upright inplace. STATE_3 23 could be the state of walking in a relatively straightline and STATE_1 25 could be the state of running. The transition fromwalking to running is indicated by 24. For each time unit a person is inone of the activity states. 22 indicates the person continued walking inthe unit time. It is apparent that this simple model can be extendedappropriately to cover an arbitrary number of action states. Thestructure of such models can be made from human knowledge and heuristicsand the states and transitions can have a known interpretation, howeverthe states and transitions can also be learned directly from the datausing various techniques that can provide classifications of contactand/or pressure data that correspond to action states and identificationof state transitions, including clustering, neural networks, decisiontree ensembles, and multi-level classification approaches using one ormany types of models. In the invention the indicative contact and/orpressure patterns for states are learned from observations of objects inthat activity state, or inferred from contact and/or pressure patternmodels of other activity states, either singly or in combinations. Atraining collection of behaviors is produced by using sequences ofactions in a locomotion sequence and then manually assigning the actionsequence to a behavior, or applying some algorithm such as similarity toa reference collection of behaviors or previously observed actionsequences. This training collection provides an input to a supportvector machine to make a behavior prediction model. The invention inputsa locomotion sequence of actions and object properties to the predictionmodel and assigns a behavior to the object for the segment. It isapparent that sequences of behaviors can be used in training sets tomake predictive models that accept sequences of behaviors to predictobject behaviors as well. Intentions can be assigned to objects based onthe behaviors by using classifiers trained with data sets whereintentions have been assigned to behaviors and collections of behaviorsmanually or algorithmically from similarity to associations of intentionto behaviors in reference data sets or by similarity to observedbehaviors to which intentions have been associated. Support vectormachines are used in a specific embodiment of this aspect of theinvention, but it is apparent that other supervised machine learningtechniques can be used as well. A variant of this embodiment of theinvention uses k-mean clustering to learn groupings of action sequencepatterns in the locomotion segments to make assignments of behaviorswithout attributing interpretable class labels. In the same way suchunsupervised techniques are used to attribute a distinct class ofintention to a locomotion sequence without interpretation. It isapparent that such systems can be dynamically extended to distinguishnew types of behaviors and intentions from the input data.

In a specific embodiment of the invention, a number of pressure sensingtiles, each with a number of pressure sensing units or elements, aredeployed to cover a work area, for example a station or zone on anassembly line. The pressure data of a worker is recorded and thepressure data is analyzed using the activity state model to identify theactivity states and state transitions to characterize their workactivities as a sequence of poses and actions, for example actions toassemble a product. The analysis includes counts of assumed poses,including the number of poses of specific types, for example ones thathave certain profiles of balance that indicate stretching or awkwardstances, the number of steps and other movements, and variousstatistical and other calculations on their actions and actionsequences. One utility of such an invention embodiment is to provideobjective data to support ergonomic evaluation of a worker's actions andwork environment. The recorded work activity and pose information can berecorded for later analysis, transmitted for remote processing andanalysis and/or processed locally. The resulting analysis can be used toprovide feedback to the worker to reduce ergonomic risks, such aswork-related musculoskeletal disorders, by altering their activity. Theanalysis can be used to modify the work environment by adjusting tools,production processes or environmental conditions.

In one embodiment a Markov model of human, agent, or object movement isapplied to new object trace instances to identify changes in locomotionstate, for example stopping or turning, to segment the object locomotiontrace. A model is trained using observed segment data that has beenlabeled with human judgments of behavior and intent. Observed locomotionsegments can then be classified and behaviors and intents assigned tothe person or object. In another embodiment the model is trained usingunordered locomotion segment sequences to allow observed locomotion datato be distinguished as probably belonging to one or more behavior orintent classes. Another specific embodiment is to use a convolutionalneural network to learn the actions and/or action sequence segments fromthe input data. Yet another specific embodiment is to use random forestdecision tree ensembles to learn the actions and/or action sequencesegments. Such distinguished locomotion properties can be interpreted tosome degree by humans or systems using contextual or other informationthat is exogenous to the locomotion traces. For example, a person whoselocomotion behavior class is characterized by relatively straight lineand constant motion that is not aligned with an entrance to a store canbe interpreted as a probable instance of ‘not browsing’ behavior. Thislabel can be applied to the locomotion behavior class and used in thetraining of an improved behavior prediction model. In this way thepredictive performance of the invention can be incrementally improvedand the domain of utility can be extended.

In another embodiment, sequences of segments are used to train thebehavior classification model or as inputs to an unsupervisedclassification model. Further, the behavior and intent modeling can alsouse with input locomotion segments that have been enhanced with otherfeatures of the person, agent, or object, for example statistics of gaitparameters or activity state changes during the segment. Additionally,the contextual features of location or environmental or other conditionsexogenous to the contact and/or pressure measurements can be used. Suchmodels can be used individually or in combinations to predict and assignlabels to an input to the system that embodies the invention. Suchmodels can also be used to modify the inputs provided to classificationmodels, both in the process of creating new models and in operationalsettings to make calculations using established models. Various machinelearning approaches can be used to make such models including decisiontrees, random forests, hidden Markov models, and neural networks,including convolutional neural networks. Reinforcement learning can beused with segments and segment sequences by assigning locomotion andactivity goals and an associated utility function. Unsupervised machinelearning approaches can also be used, including clustering, associationrule mining, self-organizing maps, and dimensionality reductiontechniques, such as Principle Component Analysis, to distinguishbehaviors.

The invention can identify anomalous behaviors by determining if anobserved segment pattern is a member of a previously learned behaviorclass. One particular technique to identify anomalous behaviors usingcontact and/or pressure input data that has been processed to identifyobject paths is to use a multi-layer process of clustering andsequencing as taught in U.S. Pat. No. 8,494,222 for visual observationsof object paths. An anomalous behavior is detected when the observedpattern does not have an acceptable fit with one or more of a collectionof behavior patterns as determined by application of rules learned usingassociation rule mining techniques or by application of Markov LogicNetworks or rules set heuristically. The invention can also detectanomalous behaviors by calculating the similarity of the observedbehavior patterns to stored or calculated parameters of known behaviorpatterns or by exchangeability with generated sequences of knownpatterns, as produced, for example, using a generative processimplemented by the system, for example by using urn processes. Thedegree of similarity or exchangeability to infer anomalous or normal orexpected behavior can be set heuristically or by learning from trainingexamples using supervised or semi-supervised techniques or learneddirectly with unsupervised techniques. In comparing the observedbehavior with the reference rules and pattern classes, it is to beunderstood that the reference rules and pattern classes can includepositive and negative behavior class instances with respect to normal,expected, or anomalous behavior patterns and the interpretation of theresults of comparing the observed behavior pattern is appropriatelyadjusted. For example, an observed pattern that is measured assignificantly similar to a pattern or patterns that have been labeled asanomalous or suspicious will be interpreted as anomalous behavior aswould an observed behavior pattern that fit none of the behaviorpatterns accessible by the system. The system can generate an alertsignal to a human or another system when an anomalous behavior isdetected or the system can communicate with another system to record thebehavior and/or provide an input to monitor or make a decision, forexample to take some automated action.

The invention is able to infer social relationships between people on asurface using a technique similar to that taught for processing a seriesof visual images by U.S. Pat. No. 7,953,690. FIG. 4 illustrates aprocess by which social relationships can be inferred in an embodimentof the present invention from surface contact and/or pressure data 28.The process first locates people or agents on a surface 29 usingtechniques disclosed above, and then extracts features for each agent,including location, path and path characteristics, and other featuresincluding probable age range and gender 30. The extracted features canbe stored locally or remotely for analysis or future use 31. Thefeatures and the locomotion paths for the collection of agents can thenbe analyzed using various techniques, including calculation ofsimilarities and relationships between patterns thereof 34. Suchsimilarities and patterns may include calculation of the degree ofexchangeability between paths and subsequences of paths. Suchcalculations and analysis can be carried out on pairs or agents or onany combination of agents and the results can be stored for currentanalysis or future use 33. Association rules learned from training dataor heuristic rules or a Markov Logic Network are then applied to theidentified similarities and patterns between two or more agents to infersocial relationships 36. The inferred social relationships can be storedfor additional analysis or future use 37, or they can be provided to oneor more application programs 39 or communicated to humans or othersystems or used to decide to trigger some action, for example fashioningand/or communicating a commercial offer to people with an inferredsocial relationship. In the foregoing, it is to be understood that theinvention can infer social relationships not only for individuals, butalso for and between groups of individuals with suitable rules andapplication of the similarity and pattern comparison algorithms. Forexample, relationships between teams of agents can be inferred using thesame techniques.

It can be seen that the objects of the invention set forth in thepreceding description, are efficiently attained and that certain changescan made in carrying out the above method and construction(s) withoutdeparting from the spirit and scope of the invention. Those trained inthe art of machine learning will recognize that a variety of supervised,semi-supervised and unsupervised modeling techniques can be used aloneor in combinations to process the data inputs, perform the higher levelanalysis, and to create and apply models to accomplish the variousinvention utilities. It is intended that all matter contained in theabove description and shown in the accompanying drawings shall beinterpreted as illustrative and not in a limiting sense.

REFERENCES CITED All of which are Incorporated by Reference Herein U.S.Patent Documents

8,138,882 A * March 2012 Mai Do, et al. 340/5.1 8,494,222 B2 * July 2013Cobb, et al. 382/305, 312 7,953,690 May 2011 Luo, et al. 706/475,952,585 June 1997 Trantzas and Haas 338/47

Ep Patent Application Documents

EP20,130,177,483

1/2014 Greene

OTHER PUBLICATIONS

-   [1] Anne F Ambrose, Mohan L Noone, V G Pradeep, Beena Johnson, K A    Salam, and Joe Verghese. Gait and cognition in older adults:    Insights from the Bronx and Kerala. Ann Indian Acad Neurol, 13(Suppl    2):S99-S103, December 2010.-   [2]D. Austin, T. Leen, T. L. Hayes, J. Kaye, H. Jimison, N. Mattek,    and M. Pavel. Model-based inference of cognitive processes from    unobtrusive gait velocity measurements. In Engineering in Medicine    and Biology Society (EMBC), 2010 Annual International Conference of    the IEEE, pages 5230-5233, August 2010.-   [3] Moez Baccouche, Franck Mamalet, Christian Wolf, Christophe    Garcia, and Atilla Baskurt. Sequential Deep Learning for Human    Action Recognition. In Proceedings of the Second International    Conference on Human Behavior Unterstanding, HBU'11, pages 29-39,    Berlin, Heidelberg, 2011. Springer-Verlag.-   [4] Alan Branzel, Christian Holtz, Daniel Hoffmann, Dominik Schmidt,    Marius Knaust, Patrick Luhne, Rene Meusel, Stephan Richter, and    Patrick Baudisch. GravitySpace: Tracking Users and Their Poses in a    Smart Room Using a 2D Pressure-Sensing Floor. In CHI 2013. ACM,    2013.-   [5] Takuya Murakita, Tetsushi Ikeda, and Hiroshi Ishiguro. Human    Tracking using Floor Sensors based on the Markov Chain Monte Carlo    Method. In ICPR (4), pages 917-920, 2004.-   [6]R. J. Orr and G. D. Abowd. The Smart Floor: A Mechanism for    Natural User Identification and Tracking. In Conference on Human    Factors in Computing Systems, pages 275-276, 2000.-   [7] Gang Qian, Jiqing Zhang, and Assegid Kidané. People    Identification Using Gait Via Floor Pressure Sensing and Analysis.    In Proceedings of the 3rd European Conference on Smart Sensing and    Context, EuroSSC '08, pages 83-98, Berlin, Heidelberg, 2008.    Springer-Verlag.-   [8] Ruben Vera Rodriguez, Richard P. Lewis, John S. D. Mason, and    Nicholas W. D. Evans. Footstep Recognition for a Smart Home    Environment. International Journal of Smart Home, 2(2):95-110, April    2002.-   [9] Axel Steinhage and Christi Lauterbach. Monitoring Movement    Behavior by Means of a Large Area Proximity Sensor Array in the    Floor. In Btörn Gottfried and Hamid K. Aghajan, editors, BMI, volume    396 of CEUR Workshop Proceedings, pages 15-27. CEUR-WS.org, 2008.-   [10] Miika Valtonen, Jaakko Mentausta, and Jukka Vanhala. TileTrack:    Capacitive Human Tracking using Floor Tiles. In PerCom, pages 1-10.    IEEE Computer Society, 2009.-   [11]F. Wang, E. Stone, M. Skubic, J. M. Keller, C. Abbott, and M.    Rantz. Toward a Passive Low-Cost In-Home Gait Assessment System for    Older Adults. Biomedical and Health Informatics, IEEE Journal of,    17(2):346-355, 2013.-   [12] William H. Warren and Brett R. Fajen. Behavioral Dynamics of    Visually-Guided Locomotion. In A. Fuchs and V. Jirsa, editors,    Coordination: Neural, behavioral, and social dynamics. Springer,    Heidelberg, 2008.

It is also to be understood that the following claims are intended tocover all of the generic and specific features of the invention hereindescribed and all statements of the scope of the invention which, as amatter of language, might be said to fall there between.

Although the invention has been described in detail in the foregoingembodiments for the purpose of illustration, it is to be understood thatsuch detail is solely for that purpose and that variations can be madetherein by those skilled in the art without departing from the spiritand scope of the invention except as it may be described by thefollowing claims.

1. A computer-implemented method to learn classifications and propertiesof one or more objects by processing a sequence of surface contactand/or pressure measurements captured by a plurality of local contact orpressure sensing system at a time and over time, the method comprisingthe steps of: receiving a frame or frames of the sequence which includesdata for a plurality of contact or pressure measurements included in theframe; identifying one or more collections of contact and/or pressuremeasurements in the frame, where each collection represents an object onthe surface; generating models to extract one or more features from thecontact and/or pressure measurement collections that are associated witheach identified object; extracting a plurality of features from thecollections of contact and/or pressure measurements; classifying each ofthe collections of contact and/or pressure measurements using a trainedor untrained classifier; supplying the extracted features and/or theobject classifications from one or more frames to a machine learningengine; and using the machine learning engine to generate values for oneor more properties of one or more objects and/or generate semanticrepresentations of the behavior of one or more objects over a pluralityof frames, where the machine learning engine is configured to learnproperties and behavior patterns observed in the contact and/or surfacepressure measurements over the plurality of frames and to identifypatterns of behavior by the classified objects.
 2. The method of claim 1further comprising the steps of locating one or more objects on thesurface by detecting one or more features using a plurality of featureprediction models applied to the entire surface or to regularly orrandomly selected regions of the surface, or by using similarity to anexemplar or to a reference collection of examples or to previouslyobserved objects, applied to the entire surface or to regularly orrandomly selected regions of the surface.
 3. The method of claim 2further comprising locating and tracking one or more objects on thesurface by detecting their current position in a space-time window andcalculating the overlap with previous detected positions or bypredicting the next position of the object or objects using learnedmodels from the configured machine learning engine or by applying asearch algorithm using the previously detected positions.
 4. The methodof claim 3 further comprising the reacquisition of one or more objectswith some level of confidence when they reenter the surface by comparingdirectly or with some feature transform their feature signature withstored models of objects accessible by the system.
 5. The method ofclaim 3 further comprising recording the location and monitoring thepaths of one or more objects on the surface stored in local memory orremotely.
 6. The method of claim 3 further comprising the segmentationof the recorded paths of one or more objects into units using learnedmodels from the configured machine learning engine or by applyingheuristic rules or rules learned from observations not based on contactor pressure data.
 7. The method of claim 5 further comprising thesegmentation of the recorded paths of one or more objects into unitsusing measurements from other systems in real time or from stored dataof the same objects or exemplars of the objects.
 8. The method of claim1 further comprising the configuration of the machine learning module toinfer properties of objects that are correlated with contact andpressure properties of objects.
 9. The method of claim 8 furthercomprising the identification of people from combinations of inferredproperties and contact and/or pressure patterns.
 10. The method of claim8 further comprising the identification of people from combinations ofinferred properties and contact and/or pressure patterns and theirlocation.
 11. The method of claim 3 further comprising the calculationof gait velocity and balance using learned models from the configuredmachine learning engine.
 12. The method of claim 11 using changes inpatterns of gait velocity and balance measurements recorded at disparatetimes to calculate the future risk of falling using learned models fromthe configured machine learning engine.
 13. The method of claim 11 usingchanges in patterns of gait velocity and balance measurements recordedat disparate times and correlating them with changes in a referencecollection of expert judgments about the future risk of falling based onsequences of gait velocity and balance measurements recorded atdisparate times.
 14. The method of claim 11 using changes in patterns ofgait velocity and balance measurements recorded at disparate times tocalculate the rate of cognitive capacity decline using learned modelsfrom the configured machine learning engine.
 15. The method of claim 11using changes in patterns of gait velocity and balance measurementsrecorded at disparate times and correlating them with changes in areference collection of expert judgments about the rate of cognitivecapacity decline based on sequences of gait velocity and balancemeasurements recorded at disparate times.
 16. An apparatus fordetermining a non-apparent attribute of an object comprising: a sensorportion with which the object makes contact and to which the objectapplies pressure; a computer in communication with the sensor portionthat receives signals from the sensor portion corresponding to thecontact and pressure applied to the sensor portion, determines from thesignals the non-apparent attribute; and an output in communication withthe computer that identifies the non-apparent attribute determined bythe computer.
 17. The apparatus of claim 16 wherein the object is aperson, the non-apparent attribute is cognitive capacity decline, thesensor portion is a plurality of sensor tiles forming a walkway, and thesignals are indicative of gait of the person moving on the walkway. 18.A method for determining a non-apparent attribute of an objectcomprising the steps of: making contact and applying pressure with anobject to a sensor portion; receiving signals by a computer incommunication with the sensor portion from the sensor portioncorresponding to the contact and pressure applied to the sensor portion;determining by the computer from the signals the non-apparent attribute;and identifying at an output in communication with the computer thenon-apparent attribute determined by the computer.
 19. The method ofclaim 18 wherein the object is a person, the non-apparent attribute iscognitive capacity decline, the sensor portion is a plurality of sensortiles forming a walkway, and the signals are indicative of gait of theperson moving on the walkway.